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Abstract 

This  article  presents  a  guidance-assistance  system  for  agricultural  machines.  It  is  based  on 
analysis  of  the  vehicle’s  environment  by  image  processing  to  deduce  a  control  law  in  the 
image  space.  Two  algorithms  of  image  processing  and  two  control  laws  are  presented.  It 
presents  some  results  of  crop  edge  detection  and  control  of  the  vehicle’s  trajectory  in  several 
conditions.  These  results  show  the  feasibility  of  such  a  project  and  the  necessity  to  know  the 
reliability  of  the  image  processing  results  to  secure  the  viability  of  the  complete  system. 
©  2000  Elsevier  Science  B.V.  All  rights  reserved. 
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1.  General  introduction 

Working  on  new  equipment  to  increase  agricultural  productivity  and  safety, 
reducing  the  difficulties  of  human  tasks,  CEMAGREF  and  LASMEA  have  devel¬ 
oped  a  guidance-assistance  system  for  agricultural  vehicles.  The  aim  of  this  system 
is  to  help  the  operator  to  guide  the  machine  in  order  to  give  more  attention  to  the 
control  of  other  functions.  To  reach  this  aim,  we  use  a  visual  perception  of  the 
environment  because  this  method  of  sensing  gives  a  solution  well-adapted  to  many 
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situations.  This  project  began  with  research  on  a  visual  guidance-assistance  system 
for  a  mower  in  1991  and  has  been  adapted  to  the  development  of  the  same  system 
for  a  combine  harvester  in  1995.  Several  works  on  guidance-assistance  systems  in 
agriculture  in  the  world  have  been  published  during  the  1990s  (Jarvis,  1990; 
Nobutaka,  1990;  Amat  et  al.,  1993;  Klassen  et  al.,  1994;  Ollis  and  Stentz,  1996, 
1997;  Billingsley  and  Schoenfisch,  1997;  Marchant  et  al.,  1997;  Sanchez  and 
Marchant,  1997).  We  particularly  notice  the  work  of  M.  Ollis  on  the  Demeter 
project  at  Carnegie  Mellon  University  (Ollis  and  Stentz,  1996,  1997).  The  most 
important  problem  in  these  applications  is  to  extract  information  about  vehicle 
environment  in  difficult  conditions  found  in  agriculture  (vegetation,  sunshine 
conditions,  relief).  So  in  the  first  part  of  this  paper,  we  present  two  environment 
perception  methods  used  to  guide  a  mower  or  a  combine  harvester.  In  the  second 
part  we  detail  two  control  laws  applied  to  agricultural  vehicles. 


2.  Image  processing 

2.1.  Introduction 

Within  the  framework  of  the  guidance-assistance  of  an  agricultural  vehicle,  and 
especially  along  a  crop  edge,  CEMAGREF  and  LASMEA  have  amassed  important 
experience  for  many  years.  Thus,  the  brief  comments  which  follow  concerning  the 
problematic  characteristics  of  a  vision  system  in  the  agricultural  environment, 
specially  in  relation  to  plant  cover,  partly  explain  the  research  orientation  of  the 
team  at  Clermont-ferrand  (Derras  et  al.,  1994).  The  first  stage  of  this  work 
consisted  of  experiments  needed  to  characterize  the  environment.  The  acquisition  of 
an  important  bank  of  images  has  made  it  possible  to  measure  the  effect  of  different 
parameters  such  as  the  variation  of  the  sunshine  conditions,  the  density  and  the 
type  of  crop  cover.  The  first  observation  from  these  studies  was  the  significant 
variability  of  image  luminance.  So  we  have  chosen  both  parameters  of  luminance 
and  texture  as  the  best  criteria  for  characterizing  images.  The  second  observation 
was  the  undesirable  effect  of  the  shadow  generated  by  the  vegetation,  particularly 
the  appearance  of  a  dark  area  between  the  cropped  zone  and  the  uncropped  zone. 


Fig.  1.  Problem  of  outdoor  vision:  three  images  with  different  positions  of  the 
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Fig.  1  illustrates  the  previous  point.  In  the  left  picture,  the  acquisition  was  done  in 
a  configuration  where  the  sun  was  positioned  behind  the  sensor.  In  this  case,  the 
luminance  is  a  good  parameter  in  order  to  extract  the  position  of  the  crop  edge.  In 
the  middle  picture,  in  which  the  only  modification  was  the  position  of  the  sun  (in 
front  of  the  sensor),  there  is  a  notable  absence  of  contrast.  In  this  case,  we  have  to 
use  texture  parameters  so  as  to  find  the  crop  edge.  The  right  picture  is  an  example 
of  a  shadow  caused  by  the  vegetation.  To  segment  these  images,  we  use  two 
different  methods:  the  first  is  an  unsupervised  segmentation  by  Markov  fields  and 
the  second  is  a  supervised  method  which  employs  a  reliability  criterion  in  the 
segmentation. 

2.2.  Example  of  unsupervised  method 

To  be  independent  of  the  region  number  in  the  image  and  the  many  situations 
found  in  a  natural  environment,  we  have  developed  an  unsupervised  segmentation 
algorithm  using  Markov  fields  (Derras  et  al.,  1994). 

2.2.1.  Parameters 

As  we  have  explained  in  the  introduction  of  this  section,  two  types  of  parameters 
made  up  the  input  data  of  the  Markov  fields: 

•  two  grey  level  parameters  which  are  the  maximum  of  the  histogram  and  the 

second  moment, 

•  two  texture  parameters  which  are  the  entropy  and  the  homogeneity. 

In  order  to  chose  these  four  descriptors,  we  have  tested  many  image  parameters. 
We  mainly  used  the  Haralick  method  (Haralick,  1979)  based  on  co-occurrence 
matrices,  in  which  the  principal  aim  is  to  provide  a  good  representation  of  the 
concept  of  grey  level  interactions.  The  constraints  of  a  final  application  in  a  real 
time  system  has  been  integrated  (sampling  period:  100  ms)  into  the  choice  of  the 
parameters.  This  explains  why  we  have  only  considered  parameters  which  need  a 
low  algorithm  complexity.  Let  us  consider  /,  an  image  with  a  size  of  I*Iy  pixels.  It 
can  be  divided  into  n*m  sites  with  a  size  of  Sx  =  4/ m  and  Sy  —  Iy/n.  Let  Pu  P2,  P2, 
P4,  be  four  matrices  with  a  size  of  n*m,  which  represent  the  maximum  of  the 
histogram,  the  second  moment,  the  homogeneity  and  the  entropy,  computed  for 
each  site.  Fig.  2,  (where  4  =  512,  4=128,  m  =  32  and  n  =  8;  so  Sx  =  Sy=  \  6) 
illustrates  this  point.  Fig.  2(a)  is  an  example  of  an  mowed/unmowed  image.  Fig. 
2(b-e)  are  grey  levels  representation  of  the  four  parameters.  In  this  example,  we 
can  see  that  only  two  parameters  seem  to  be  good  for  the  estimation  of  the  crop 
edge  (the  maximum  of  the  histogram  and  the  entropy).  But,  statistically,  all  the 
parameters  are  useful  to  segment  the  natural  images  met  in  our  applications. 

2.2.2.  Markov  segmentation,  analysis  and  choice  of  the  crop  edge 

Concerning  the  segmentation  step,  the  use  of  Markov  fields,  and  especially  of  the 
ICM  (Iterated  Conditional  Mode)  algorithm  (Cohen  and  Cooper,  1987)  gives  good 
results.  For  instance  in  Fig.  3,  we  can  see  five  regions  which  make  it  possible  to  find 
the  crop  edge.  A  list  of  all  possible  limits  is  extracted  by  a  contour  analysis  module. 


3.  Segmentation  result. 
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The  last  part  consists  of  the  choice  of  the  good  limit  in  the  list  and  its  modeling  by 
a  straight  line  whose  polar  coordinates  are  ( p  and  9).  It  is  made  by  minimization 
of  an  Euclidean  distance  criterion  between  all  possible  paths  of  the  list  and  the 
result  of  the  previous  image.  This  last  part  of  the  algorithm  is  empirically  right  but 
wrong  detections  can  appear  resulting  from  shadow  or  from  the  tire  tracks  of  the 
previous  row.  The  theory  and  the  implementation  on  a  Real  Time  architecture  are 
developed  in  (Derras  et  al.,  1991;  Derras,  1993). 

2.2.3.  Implementation  and  tests 

Implemented  in  a  real  time  system  based  on  two  68060  microprocessors,  the  crop 
edge  algorithm  detection  has  an  average  time  of  execution  which  is  about  100  ms 
per  image  (in  1997).  Nevertheless  the  segmentation  by  Markov  fields,  which  is  not 
determinist,  leads  to  the  need  for  time  limited  security  in  order  to  avoid 
possible  oscillations.  This  aspect  appears  as  an  important  disadvantage  in  a  real 
time  based  mode.  One  of  the  conclusions  of  our  tests  of  this  approach  concerns  the 
importance  of  the  reliability  of  each  crop  edge  estimation.  In  spite  of  a  high 
percentage  of  good  estimations  (90%),  determined  by  a  human  operator,  the 
guidance  is  not  reliable  enough.  As  a  matter  of  fact,  we  have  to  be  able  to  quantify 
the  uncertainty  for  each  estimation  in  order  to  isolate  the  bad  detection  of  the 
perception  module. 

2.3.  Supervised  algorithm 

On  the  assumption  that  the  scene  contains  only  two  classes  (a  cut  and  an 
uncut  one),  we  can  define  a  typical  model  of  the  image.  The  methods  which 
use  this  model  seem  to  be  weakest,  in  contrast  to  the  type  of  specific  pictures 
than  the  algorithm  based  on  the  Markov  fields  segmentation.  On  the  other 
hand,  it  is  possible  to  quantify  the  similarity  of  the  scene  with  this  model  in 
order  to  estimate  the  reliability  of  the  detected  limit.  This  point  is  essential 
and  more  important  than  the  global  performances  of  the  algorithm  so  as  to 
provide  a  good  guidance  in  an  iterative  structured  environment  (the 
current  environment  is  structured  by  the  previous  trajectory  of  the  machine). 
The  supervised  method  presented  in  this  part  provides  both  the  position  of 
the  crop  edge  and  the  reliability  criterion  associated.  It  can  be  divided  into  three 
parts: 

•  a  correlation  function  between  the  model  and  the  real  image  is  perform  for  each 
parameter, 

•  each  correlation  function  is  analyzed  in  order  to  quantify  the  reliability, 

•  the  theory  of  evidence  (Shafer,  1976)  makes  it  possible  to  merge  the  information 
collected  after  the  transformation  of  the  correlation  function  into  a  mass 
function.  The  position  of  the  crop  edge  is  calculated  through  a  maximum  of 
belief  rule.  An  estimation  of  a  reliability  criterion,  associated  with  the  detected 
position  of  the  crop  edge  is  performed  and  can  be  used  in  order  to  validate  the 
result  (Chateau  et  al.,  1997). 
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2.3.1.  The  correlation  approach 

We  assume  that  the  straight  line  model  of  the  crop  edge  is  a  nearly  vertical  line 
in  the  image  plane.  So,  we  sum  by  row  each  matrix  Pjje  [1,  4]  <=  N  and  we  obtain 
a  vector  Lhje[\,  4]  <=  /V  (each  Lj  has  a  dimension  of  m,  and  m  is  the  Pj  column 
number;  like  it  is  calculated  in  Section  2.2.1)  in  which  the  crop  edge,  materialized 
by  a  step  transition,  appears  with  less  noise  than  in  the  2D  matrix.  This  transforma¬ 
tion  is  realized  by  the  following  equation: 

L^Um.Pj)  (1) 

where  M  =  '{  1,  .*.  . ,  1 )  representing  a  vector  of  n  elements  (n  is  the  number  of  sites). 
Fig.  4  is  a  representation  of  L,  in  the  example  of  the  Fig.  2.  The  estimation  of  the 
position  of  the  step  can  be  performed  by  computing  the  deviation  for  each  vector 
Lf,  but  this  method  is  not  adapted  to  noisy  signals.  W.Y.  Liu  proposed  an  operator 
for  the  detection  of  transitions  in  a  very  noisy  signal.  It  is  based  on  the  local 
properties  of  stationarity  of  the  signal  (Liu,  1994).  The  approach  described  here  is 
based  on  performing  a  correlation  function  computing  between  vector  Ly  and  a 
reference  vector  G  (a  step  transition).  The  correlation  function  between  the  two 
vectors  is  defined  by  (Max,  1985): 

Convex)  =  |  Lj(k).G(k  -  x )  (2) 

Let  us  define  the  reference  vector  G  by: 


with  /?e[0,  0.5]  c  R  and  tx  =  1  or  —  1. 

We  define  tx  as  the  crop  edge  characteristic  coefficient.  It  is  an  a  priori  component 
of  the  algorithm.  In  our  application,  tx  =  1  and  /?  =  5/m. 


LI :  max  of  histogram 
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We  have  to  remove  the  mean  value  of  Lj  before  computing  the  correlation 
function  between  the  measured  signal  and  the  sought  pattern.  The  position  of  the 
maximum  of  the  function  Corri  G  is  the  most  probable  position  of  the  search 
pattern  in  the  initial  signal.  We  can  deduce  the  position  of  the  crop  edge  from  this 
information.  Fig.  5  is  a  representation  of  the  correlation  function  computing  from 

l2. 

2.3.2.  Data  fusion 

The  previous  part  describes  a  method  which  performs  some  correlation  functions 
associated  with  the  luminance  and  texture  parameters.  Now,  all  these  functions 
must  be  merged  in  order  to  estimate  a  more  probable  position  of  the  crop  edge. 

2.3.2.I.  Estimation  of  the  uncertainty.  If  /is  a  perfect  image  with  two  parts  and  no 
noise,  the  correlation  function  presents  only  one  maximum  which  the  value  is 
jua/m  =  2./j  with  jia  =  max(Corr/  G(x)),  and  the  quality  of  the  estimation  must  be 
maximum  (1  if  the  quantification  interval  is  [0,  1]).  Two  types  of  perturbations 
appear  in  the  image: 

•  The  image  has  at  least  three  zones  (at  least  two  possible  crop  edges).  The 
associated  correlation  function  has  the  same  number  of  maxima  than  the  number 
of  possible  crop  edges.  We  define  a  ‘ global  adequacy  coefficient’  F  =  l/q  where  q 
is  the  number  of  maxima  of  the  correlation  function  for  which  the  value  is  higher 
than  e.fia  (£e[0,  1])  is  a  sensibility  coefficient). 

•  Gaussian  noise  appears  in  the  image.  The  more  significant  the  noise  is,  the  more 
the  value  of  the  maximum  of  the  correlation  function  is  weak.  We  define  a  ‘local 
adequacy  coefficient’  A  =  fia/( 2/?)e[0,  1]). 

Both  the  local  adequacy  coefficient  and  the  global  adequacy  coefficient  are 
important  in  order  to  quantify  the  quality  of  the  estimation  of  the  crop  edge 
position. 
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23.2.2.  The  theory  of  evidence.  Two  principal  theories  can  be  distinguished  in  data 
fusion.  The  probability  models  represent  the  information  into  densities  of  probabil¬ 
ity  (Haton,  1991).  These  models  are  however  limited  when  some  conflicting 
opinions  must  be  merged  or  when  the  a  priori  information  is  unknown  in  the  case 
of  Bayes  theory.  Models  based  on  the  theory  of  evidence  (Shafer,  1976;  Bloch, 
1996)  are  adapted  for  the  estimation  of  the  imprecision  and  the  uncertainty. 

We  define  ©  as  the  set  of  the  possible  positions  of  the  crop  edge:  ©  =  {Iu...,  Im} 
where  It(ie{\, . .  . ,  m})  is  the  hypothesis:  ‘the  crop  edge  is  a  vertical  line  which 
passes  by  the  image  sites  number  i’.  In  the  probability  theory,  we  can  only  assign 
probability  measures  to  elementary  subsets  (//,).  In  the  theory  of  evidence,  belief 
can  be  assign  to  all  the  subsets  of  ©  (2®  subsets).  So,  uncertainty  can  be  set  if  we 
assign  belief  to  ©.  We  can  define  m:2®->[0,  1]  called  mass  function  whenever: 

m(0)  =  O  X  m(A)  =  1  (4) 

The  quantity  m(A)  is  called  A ’s  basic  probability  number  (basic  belief  or  mass 
given  to  A),  and  it  is  understood  to  be  the  measure  of  the  belief  that  is  committed 
exactly  to  A.  It  is  possible  to  combine  mass  functions  by  the  orthogonal  sum 
operator:  consider  two  independent  sources  of  information  represented  by  their 
mass  functions  ml  and  m2.  Then  m]  and  m2  can  be  combined  by: 


ml2(z)  =  ml(z)®m2(z)  =  K*  £  mfX)*m2(Y)  (5) 

iX.Y)jXr\  Y—z 

where  z  is  a  subset  of  ©,  X  and  Y  are  different  parts  of  the  frame  of  discernment 
and  with: 


-  =  1  -  X  mfX)*m2(Y) 

A  (X,  Y)jXr\  Y=0 


(6) 


The  degree  of  conflict  between  the  two  sources  can  be  evaluated  with  K.  Before 
building  the  mass  functions,  let  us  compute  probability  distributions  from  the 
correlation  functions: 


Cj(x)  =  rot,./; 


[Corr, 


;(x)  if  Corr/  G(v)  >  0 
0  Otherwise 


Pj(x)  =  — 

I 


- Cj(x) 

,CJk) 


(7) 


where  rotr/;  is  the  operator  of  right  rotation  of  /?  terms.  The  aim  of  this  transforma¬ 
tion  is  to  remove  the  offset  created  by  the  position  of  the  transition  in  the  reference 
vector,  pj  is  a  probability  density  function  associated  with  the  parameter  Pjm  Let  us 
define  four  mass  functions  mu  m2,  m3,  mA  where: 

/»,(©)  =  1  -  Fj.Aj  m/x )  =  (1  -  m/0)).p/x) 


(8) 
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where  wy(0)  is  the  residual  uncertainty  of  the  estimation,  and  (Fp  Ay)  are  the  global 
and  local  adequency  coefficient  associated  with  the  parameter  Pj.  It  is  now  possible 
to  compute  the  global  mass  using  the  Dempster  rule  of  combination  defined  in  Eq. 
(5): 

m  =  ml®m2®3m4  (9) 

The  value  of  m(0)  is  a  good  estimation  of  the  reliability  of  the  detection  of  the 
crop  edge.  The  maximum  of  the  mass  function  m  indicates  the  most  probable 
position  of  the  crop  edge. 

Fig.  7  is  an  example  of  a  crop  edge  detection  in  a  image  of  grass  vegetation.  The 
low  level  algorithm  computes  four  tables  of  parameters  which  are  the  same  as  used 
in  the  unsupervised  method.  We  then  compute  four  correlation  functions  between 
each  parameter  and  the  image  model.  Subsequently,  these  four  correlation  func¬ 
tions  are  transformed  into  four  mass  functions  (Fig.  6).  We  can  notice  that  the 
value  of  the  residual  uncertainty,  m(0),  is  more  important  for  the  second  moment 
and  the  homogeneity  which  are  in  this  example,  the  two  less  representative 
parameters.  The  composed  mass  function  can  be  computed  by  the  Dempster’s  rule 
of  combination.  Results  show  that  the  maximum  of  belief  is  given  for  the  position 
of  the  limit  on  the  site  number  19  with  a  value  of  residual  uncertainty  about  0.13. 
The  right  part  of  the  Fig.  7  shows  the  final  crop  edge  estimation  calculated  by  the 
global  algorithm.  If  all  the  parameters  are  unrepresentative,  consequently  the  value 
of  the  global  residual  uncertainty  will  be  high.  If  we  set  a  threshold  on  it,  it  is 
possible  to  validate  (or  not)  the  estimation  of  the  crop  edge. 


Fig.  6.  Mass  functions  performed  for  the  four  parameters. 
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2.4.  Discussion 

If  we  have  good  results  of  image  segmentation  with  the  unsupervised  method 
thanks  to  the  use  of  texture  parameters  and  Markov  fields,  it  is  sometimes  difficult 
to  find  the  good  limit  and  we  have  no  idea  of  the  residual  uncertainty  of  its 
position.  The  supervised  algorithm  has  a  principal  characteristic:  The  use  of  the 
reliability  criteria  associated  with  each  parameter  is  very  important  into  the  data 
fusion  module.  The  estimation  of  a  global  criteria  for  the  more  probable  position 
of  the  crop  edge  provides  a  control  of  the  presence  of  the  crop  edge.  In  outdoor 
vision,  where  the  quality  of  the  acquisition  depends  on  the  relative  position 
camera/sun,  and  where  the  number  of  types  of  pictures  are  very  high,  control  of  the 
reliability  of  the  detection  is  essential.  The  guidance  cannot  be  done  correctly 
without  this  module.  The  vehicle  is  driven  automatically  via  a  closed  servo-control 
loop.  So,  if  the  estimation  is  bad  during  four  or  five  iterations,  the  crop  edge  can 
leave  the  image  window  and  the  guidance  system  will  fail.  If  the  algorithm  detects 
the  wrong  estimation,  it  is  possible  to  initially  set  the  guidance-assistance  system  in 
a  prediction  mode,  then  in  an  alarm  mode,  so  as  to  return  control  of  the  vehicle 
temporally  to  the  driver. 


3.  Control  laws 

The  guidance-assistance  system  sets  two  problems.  The  first  one  is  the  perception 
of  the  environment,  the  second  one  is  the  vehicle  control.  The  consideration  of 
these  two  difficulties  shows  the  link  between  the  second  and  the  first  problem.  We 
add  a  feedback  loop  which  makes  it  possible  to  correct  errors  and  drifts  of  the 
system  due  to  imprecise  modeling  and  measures.  Fig.  8  gives  a  system  diagram  of 
the  application.  From  the  visual  information  (0  and  p,  which  are  the  coordinates  of 
the  crop  edge  in  the  image  space),  given  by  the  image  processing  algorithm,  we  have 
designed  well-adapted  control  laws  for  our  applications.  First,  simple  control  laws 
based  on  a  kinematic  model  of  our  machines  have  been  developed.  We  present  two 
of  them  which  are  tested  on  two  vehicles  (a  mower  and  a  combine  harvester).  The 
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first  one  needs  scene  coordinates  and  uses  the  results  of  Kehtarnavaz  in  the  BART 
project  (Kehtarnavaz  et  al.,  1991).  The  second  control  law  is  based  on  the  visual 
servoing  theory  (Espiau  et  al.,  1992)  and  uses  only  the  coordinates  given  by  the 
image  frame.  It  has  been  developed  for  robot  manipulators  and  we  have  adapted  it 
for  a  non-holonomic  mobile  robot.  In  the  two  cases  we  do  not  need  a  dynamic 
model  of  our  machines  because  of  the  low  speed  of  agricultural  machines  ( < 
10  km/h). 

3.1.  Control  law  with  3D  trajectory 

In  this  part,  we  present  the  definition  of  a  trajectory  which  allows  a  vehicle  to  go 
from  its  initial  position  to  a  new  position  in  front  of  it.  We  have  tested  this 
approach  with  a  mower  using  application  parameters  (distance  from  image  center 
and  machine  position,  camera  height...)  which  are  adapted  to  a  mower.  However, 
the  presented  method  is  only  dependent  on  the  kinematic  model  of  the  machine  and 
can  be  extended  to  other  vehicles  (tractor,  combine  harvester...).  Our  application 
uses  a  video  sensor,  so  we  have  to  eliminate  all  large  movements  of  the  vehicles  to 
keep  the  crop  edge  in  the  image.  To  do  that,  the  trajectory  calculated  needs  to  go 
from  the  initial  position  to  the  desired  position  with  a  large  radius  of  curvature.  The 
idea  is  to  find  two  tangent  circles,  one  of  which  includes  the  current  position  of  the 
machine  and  the  other  the  desired  position  (Fig.  9).  Kehtarnavaz  in  (Kehtarnavaz 

1 . Perception . )<— ]_ . Sensor . . [<~C  Environment  [ 


I  Model ~  ZNS»1Z  Control  ZHZ  Vehicle  | 

Fig.  8.  General  control  diagram. 
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Fig.  9.  Vehicle  trajectory. 


Fig.  10.  Simulation  conditions  and  response  to  a  step  of  1  m. 


et  al.,  1991)  uses  circles  with  the  same  radius  of  curvature.  So  the  radius  of 
curvature  is  same  in  magnitude  along  all  the  trajectory.  In  our  case  we  use  another 
condition  (Debain  et  al.,  1994)  which  makes  possible  to  maximize  the  radius  of 
curvature.  Then  the  trajectory  followed  by  the  machine  is  smoother  and  better  for 
the  image  processing  module.  Using  this  control  law,  we  have  simulated  the 
machine  behavior  at  a  speed  of  3  km/h.  Fig.  10  gives  the  response  on  a  trajectory 
with  a  step  of  1  m.  This  curve  shows  a  good  control  of  the  machine  which  corrects 
the  error  of  1  m  in  less  than  10  s.  It  also  shows  the  method’s  stability  because  we 
do  not  notice  any  oscillation  or  overshoot.  Some  tests  in  real  conditions  have 
shown  the  same  behavior  as  in  simulation  with  a  precision  better  than  10  cm.  In 
this  experiment,  the  camera  is  embedded  on  the  mower  (camera  height  =  100  cm, 
camera  angle  =  10°). 
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3.2.  Control  law  in  the  image  space 


Our  vehicles  often  work  on  bumpy  ground  which  induces  a  lot  of  noise  in  the  3D 
point  coordinates  estimated  from  the  image  information.  So  we  tried  another 
control  law  which  uses  only  image  information.  It  has  been  developed  for  a  farm 
robot.  Many  works  concerning  robot  manipulators  (Feddema  et  al.,  1989;  Espiau, 
1993;  Hashimoto,  1993;  Corke,  1996;  Hager  et  al.,  1996;  Chaumette,  1998)  have 
shown  the  robustness  of  this  control  in  relation  to  calibration  errors  and  noise. 
However  the  visual  servoing  approach  was  not  applicable  to  non-holonomic 
machines.  So  in  our  work,  we  have  shown  how  we  can  use  visual  servoing  to 
control  the  trajectory  at  constant  speed  of  non-holonomic  agricultural  vehicle  in  the 
image  space  (Debain,  1996;  Khadraoui  et  al.,  1998;  Martinet  et  al.,  1998).  From  the 
interaction  matrix  and  the  task  function  approach  developed  by  Espiau  and 
Samson  (Samson  et  al.,  1991),  we  deduced  a  control  law  which  gives  the  camera 
rotation  as  a  function  of  the  detected  line  coordinates  in  the  image  frame: 


(10) 


with: 

•  a,  angle  between  the  ground  plan  and  the  camera  axis; 

•  h,  camera  height; 

•  p  and  9,  coordinates  of  the  detected  line  in  the  image  space; 

•  k  and  ft  are  gains  which  depend  on  the  vehicle  model  and  the  desired  response 
curve. 

•  p*  and  6*  are  the  polar  coordinates  of  the  detected  line  when  the  machine  is 
well-controlled  (reference  line  in  image  space). 


3.2.1.  Application 

We  use  this  formalism  to  control  the  trajectory  of  a  combine  harvester.  To  do 
that,  we  search  a  kinematic  model  (Fig.  11)  of  the  vehicle  described  in  (Debain, 
1996;  Khadraoui  et  al.,  1998).  Then  we  find  the  equation  which  gives  the  control 
values: 


(11) 


with: 

•  rmoy,  vehicle  mean  speed; 

•  L,  distance  between  the  axles; 

•  v  and  3,  vehicle  control  variables. 

So  we  have  a  relation  between  the  control  variables  of  the  vehicle  and  the  polar 
coordinates  of  the  detected  line  given  in  the  image  frame.  It  should  be  noticed  that 
we  consider  that  the  speed  of  the  machine  is  constant  and  non-zero. 
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3.2.2.  Simulation  tests 

First  we  make  simulation  tests  in  which  the  machine  has  to  reach  a  line  1  m  away 
(Fig.  12).  Like  for  the  mower  experiments,  the  camera  is  embedded  on  the  combine 
(camera  height  =  3  m,  camera  angle  =  25°).  We  have  several  test  conditions  which 
makes  possible  to  adjust  our  gains  according  to  the  desired  curve  of  response  (Fig. 
13).  For  our  simulations,  we  choose  a  speed  of  4  km/h  and  the  sampling  period  is 
about  100  ms.  On  the  curve,  we  see  the  machine  error  between  its  current  position 
and  the  line  position.  From  t  =  l  s  we  can  see  the  response  to  the  step  which  is 
about  20  s. 


3.2.3.  Tests  in  real  conditions  with  the  machine 
After  the  simulation  tests  it  was  interesting  to  make  real  tests.  To  do  that,  we 
decided  to  follow  a  white  line  on  the  ground  to  optimize  the  precision  of  the  visual 

Axle  which  replaces  the  machine 


Fig.  11.  Vehicle  model. 
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Fig.  12.  Simulation  and  real  test  conditions. 
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Fig.  13.  A  real  test  at  4  km/h. 

Camera  which  measures  the  distance 


perception  algorithm,  because  it  is  easier  to  detect  a  white  line  than  a  limit  between 
mowed  and  unmowed  vegetation.  We  have  made  tests  in  various  conditions  (with 
or  without  additional  noise:  20%)  at  different  speeds  and  with  a  sampling  period 
which  changes  with  time. 

The  quality  of  our  control  servoing  can  be  calculated  in  real  time  by  using  a 
second  camera  which  is  continuously  measuring  the  distance  between  the  vehicle 
and  the  white  line  (Fig.  14). 

The  curve  is  done  at  a  speed  of  4  km/h  and  a  sampling  period  of  100  ms  which 
represents  the  image  processing  calculation  time  (Fig.  13).  For  the  simulation  needs, 
we  introduce  a  delay  corresponding  to  the  image  processing  calculation  time.  We 
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can  thus  compare  the  simulation  and  the  real  curve  which  are  done  in  the  same 
experimental  conditions.  We  can  notice  that  the  real  and  simulation  curves  are 
superimposed.  The  response  time  is  about  20  s  and  there  is  not  any  overshoot. 
There  is  a  lot  of  noise  on  the  real  curve  because  the  camera  which  measures  the 
distance  between  the  vehicle  and  the  white  line  vibrates  when  the  machine  runs.  The 
same  image  processing  algorithm  is  used  to  control  the  machine  and  to  measure  the 
error.  This  algorithm  has  a  precision  better  than  10  cm.  So  the  most  part  of  the 
noise  is  produced  by  the  camera  and  the  image  processing  algorithm  which  measure 
the  distance  between  the  vehicle  and  the  white  line. 


3.3.  Visual  servoing  in  difficult  conditions 

Agricultural  vehicles  often  work  on  sloping  ground.  So  we  tested  our  visual 
servoing  control  law  on  this  condition  in  simulation  and  real  tests.  In  Fig.  15  we 
can  observe  the  machine  behavior  on  sloping  ground  with  a  classical  guidance-assis¬ 
tance  system  using  visual  servoing.  The  machine  has  to  follow  a  white  line  drawn 


Fig.  15.  Automatic  driving. 
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Fig.  16.  Distance  between  the  machine  and  the  line. 
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Fig.  17.  Manual  driving. 

on  the  ground.  Fig.  16  gives  the  distance  between  the  machine  position  and  the 
white  line.  When  the  slope  appears  (at  t  =  20  s)  there  is  a  constant  error  in  the 
position  of  the  machine.  We  can  also  see  that  the  machine  is  not  parallel  to  the 
white  line  (Fig.  15).  There  is  an  angle  between  the  direction  of  the  white  line  and 
the  machine  orientation.  And  we  can  say  that  this  angle  depends  on  the  slope  and 
the  sliding  conditions.  With  the  same  experimental  conditions,  and  if  the  machine 
is  driven  by  a  person,  we  observe  an  angle  between  the  white  line  and  the  machine 
orientation  (Fig.  17).  It  is  the  same  angle  as  we  observe  for  the  case  of  the  Fig.  15. 
So  we  can  say  that  there  is  a  relation  between  this  angle  and  the  slope  of  the 
ground.  In  fact,  the  driver  corrects  the  slope  effects  by  shifting  the  machine.  But 
with  the  guidance-assistance  system  there  is  a  constant  error  at  the  end  of  the  cutter 
bar  divider  (a  tool  of  the  combine)  in  the  case  of  a  combine  harvester,  due  to  the 
slope  and  the  sliding  conditions.  Our  idea  is  to  read  the  slope  effects  through  the 
angle  between  the  machine  orientation  and  the  white  line.  We  have  to  find  a  link 
between  this  angle  and  the  constant  error  observed  with  the  guidance  system.  To  do 
that,  it  is  interesting  to  note  that  the  position  of  the  detected  line  is  not  the  same 
in  the  two  cases.  When  the  machine  is  correctly  guided  along  the  white  line  (Fig. 
17),  the  reference  line  is  not  superimposed  on  the  detected  line.  So  the  reference  line 
which  is  the  line  detected  when  the  machine  is  correctly  guided  is  not  the  same  for 
sloping  ground  (Fig.  18,  right)  and  a  flat  ground  (Fig.  18,  left).  We  have  shown  that 
the  reference  lines  are  not  the  same  if  the  machine  runs  on  flat  ground  or  sloping 
ground.  Then  we  have  noticed  that  the  slope  effect  on  the  vehicle  behavior  can  be 
described  by  an  orientation  of  the  machine.  To  find  the  set  of  all  the  reference  lines 
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Fig.  20.  Set  of  the  2D  parameters  for  each  slope  value. 


which  give  the  coordinates  of  the  detected  line  in  the  image  when  the  machine  is 
correctly  controlled,  we  have  recorded  the  image  parameters  {p,  8)  for  several  slope 
values  (Fig.  19).  The  graph  of  these  parameters  is  given  in  Fig.  20.  It  is  a  straight 
line  and  the  point  (0,  0)  represents  the  coordinates  of  the  reference  line  for  a  flat 
ground. 

Then  the  difficulty  is  to  find  the  coordinates  of  the  reference  line  that  represents 
the  current  situation  (the  slope  of  the  ground  and  the  sliding  conditions).  To  do 
that,  we  use  the  orientation  of  the  vehicle  which  is  the  consequence  of  the  slope 
effects.  Then,  with  this  orientation  we  obtain  the  coordinates  of  the  reference  line 
(Fig.  20).  So  we  obtain  new  values  for  the  input  of  the  control  law  (padapt,  8). 

Our  control  law  is  a  vision  based  control  law.  Many  authors  use  a  proportional 
vision  based  control  law  (Chaumette,  1990;  Corke,  1993;  Khadraoui  et  al.,  1996). 
Due  to  the  slope  effect,  an  integrator  module  becomes  necessary  in  the  control  law. 
In  addition,  with  this  module,  it  is  possible  to  have  a  constant  steering  angle  even 
if  the  error  is  equal  to  zero.  The  details  of  the  proportional  integral  law  calculation 
is  given  in  (Debain,  1996).  We  just  give  the  result,  that  means  the  equation  between 
the  steering  angle  of  the  vehicle  and  the  visual  information: 


6  = 


sin  a  cos  a 
1  +h2 


n  h2  +  cos2  tx( 

®  "l  \  +h2  l  ^adapt 


*)] 


(12) 


where  A,  [1  and  kt  are  three  gains  which  control  the  behavior  of  the  vehicle  and  (p, 
8)  are  the  polar  coordinates  of  the  line  detected  in  the  image. 

Fig.  21  gives  a  view  of  the  visual  guidance  system  which  controls  an  agricultural 
vehicle  on  sloping  ground.  With  a  such  corrector  we  have  good  results  in  simulation 
and  real  conditions.  For  example,  in  the  Fig.  22  we  can  see  that  when  we  use  the 
corrector  (at  t  =  50  s),  we  correct  the  slope  effect  in  5  s.  Fig.  23  shows  the  control 
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23.  Result 
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of  a  combine  harvester  on  sloping  ground.  We  can  see  the  angle  between  the 
machine  and  the  white  line  it  has  to  follow.  We  observe  that  the  vehicle  is  correctly 
controlled  with  a  precision  better  than  10  cm.  In  these  two  examples  the  slope  is 
about  20%.  So  we  found  a  solution  to  guide  a  mobile  robot  on  sloping  ground  by 
using  only  the  sensor  of  the  visual  guidance  system.  To  do  that  we  make  the 
hypothesis  that  the  machine  has  to  follow  a  line  locally  approximated  by  a  straight 
line.  For  many  agricultural  applications,  this  hypothesis  seems  to  be  sufficient.  To 
correct  the  sloping  ground  effects  we  use  only  the  machine  orientation  which  gives 
us  the  new  steering  angle.  We  made  tests  in  simulation  and  real  conditions.  The 
results  show  a  good  behavior  of  the  machine  which  is  robust  from  different  slopes 
and  sliding  conditions. 


4.  Summary  and  conclusion 

In  this  article,  we  present  different  works  we  have  done  on  a  guidance-assistance 
system  for  two  agricultural  vehicles.  We  show  different  difficulties  found  in  our 
projects.  Firstly  the  environment  perception  described  by  means  of  image  process¬ 
ing  algorithms  which  use  supervised  or  unsupervised  methods.  In  this  part  we 
notice  the  need  to  qualify  the  reliability  of  the  perception  information  in  order  to 
guide  correctly  the  vehicles.  To  do  that,  we  give  a  solution  using  the  theory  of 
evidence  which  formalizes  the  uncertainty  and  computes  a  reliability  criterion 
associated  with  the  estimation  of  the  crop  edge.  Secondly,  two  control  laws 
determine  the  orders  of  the  machine  steering  angle  to  guide  them  along  the  crop 
edge.  The  first  one  consists  to  generate  a  reference  trajectory  and  the  second  one, 
which  is  calculated  in  the  image  space,  can  be  used  in  very  difficult  situation  like 
sloping  ground.  The  result  satisfies  most  situations  and  the  precision  of  the  system 
is  better  than  10  cm.  Nevertheless,  there  are  still  very  difficult  situations  in  which  it 
is  impossible  to  correctly  guide  agricultural  vehicles  (sun  direction  at  the  end  of  the 
afternoon,  shadow  of  the  machine  in  the  image,  lack  of  vegetation).  Future  works 
will  increase  the  environment  perception  using  others  sensors  like  laser  telemeter 
and  fuzzy  based  theory  to  combine  laser  and  video  information.  We  will  also  work 
on  guidance-assistance  system  by  CPDGPS  and  new  control  algorithms  in  3D 
space. 


References 

Amat,  J.,  Battle,  J.,  Fuertes,  J.M.,  Martinez,  A.,  1993.  Vision  controlled  robot  for  agricultural 
application.  Proceedings  of  the  24th  ISIR,  pp.  537-542. 

Billingsley,  J.,  Schoenfisch,  M.,  1997.  The  successful  development  of  a  vision  guidance  system  for 
agriculture.  Comput.  Electron.  Agric.  16,  147-163. 

Bloch,  I.,  1996.  Incertitude,  imprecision  et  additivite  en  fusion  de  donnees:  point  de  vue  historique. 
Traitement  du  signal  13,  268-288. 

Chateau,  T.,  Berducat,  M.,  Bonton,  P.,  1997.  An  original  correlation  and  data  fusion  based  approach 
to  detect  a  reap  limit  into  grey  level  image.  In:  Proc.  IEEE  Int.  Conf.  Intelligent  Robots  and  Systems, 
vol.  3.  IROS’97.  Grenoble,  France,  pp.  1258-1263. 


C.  Debain  et  al.  /  Computers  and  Electronics  in  Agriculture  22  (2000)  29-51 


Chaumette,  F.,  1990.  La  relation  vision-commande:  theorie  et  application  a  des  taches  robotiques.  PhD 
thesis.  IRISA/INRIA-Rennes.  Rennes,  France. 

Chaumette,  F.,  1998.  Potential  problems  of  stability  and  convergence  in  image  based  and  position-based 
visual  servoing.  The  confluence  of  Vision  and  Control,  LNCIS  series,  Springer  Verlag. 

Cohen,  F.S.,  Cooper,  D.,  1987.  Simple  parallel  hierarchical  and  relaxation  algorithms  for  segmenting 
non-causal  markovian  random  fields.  IEEE  Trans.  Pattern  Analysis  and  Machine  Intelligence  19  (2), 
195-219. 

Corke,  P.,  1993.  Visual  control  of  robot  manipulators-A  review,  vol.  7  of  Robotics  and  Automated 
Systems,  ISBN  981-02-1864-6,  World  Scientific  Publishing  Co.,  pp.  1-31. 

Corke,  P.I.,  1996.  Visual  Control  of  Robots:  High  Performance  Visual  Servoing.  Research  Studies  Press, 
ISBN  0-86380-207-9,  John  Wiley. 

Debain,  C.,  1996.  Lois  de  commande  pour  le  controle  et  la  mobilite  de  machines  agricoles.  PhD  thesis. 
Universite  Blaise-Pascal,  Clermont-Ferrand,  France. 

Debain,  C.,  Derras,  M.,  Berducat,  M.,  Bonton,  P.,  Gallice,  J.,  1994.  Development  of  a  visual  guidance 
system  for  an  upkeep  robot  of  natural  environments.  In:  Proc.  Int.  Symp.  Signal  Processing, 
Robotics  and  Neural  Networks,  IMACS,  Lille,  France,  pp.  121-124. 

Derras,  M.,  1993.  Segmentation  non-supervisee  d’images  texturees  par  champs  de  Markov:  application 
a  l’automatisation  de  l’entretien  des  espaces  naturels.  PhD  thesis,  Universite  Blaise-Pascal,  Clermont- 
Ferrand,  France. 

Derras,  M.,  Berducat,  M.,  Bonton,  P.,  1991.  Image  processing  and  algorithm  merging:  real  time  control 
of  an  unkeep  machine  for  natural  space.  In:  Proc.  IARP  Workshop  on  Robotics  in  Agriculture  and 
Food  Industries,  pp.  10-18. 

Derras,  M.,  Debain,  C.,  Berducat,  M.,  Bonton,  P.,  Gallice,  J.,  1994.  Unsupervised  regions  segmentation: 
real  time  control  of  an  upkeep  machine  of  natural  space.  In:  Proc.  Eur.  Conf.  Computer  Vision,  pp. 
207-212. 

Espiau,  B.,  1993.  Effect  on  camera  calibration  errors  on  visual  servoing  in  robotics.  In:  Proc.  Third 
International  Symposium  on  Experimental  Robotics,  pp.  187-193. 

Espiau,  B.,  Chaumette,  F.,  Rives,  P.  1992.  A  new  approach  to  visual  servoing  in  robotics.  IEEE  Trans, 
on  Robotics  and  Automation,  vol.  8  (3). 

Feddema,  J.T.,  Lee,  C.S.G.,  Mitchell,  O.R.,  1989.  Automatic  selection  of  image  features  for  visual 
servoing  of  a  robot  manipulator.  In:  Proc.  IEEE  Int.  Conf.  Robotics  and  Automation,  Scottsdale, 
Arizona,  pp.  832-837. 

Hager,  G.D.,  Hutchinson,  S.,  Corke,  P.,  1996.  Visual  servo  control.  IEEE  Int.  Conf.  Robotics  and 
Automation.  Minneapolis  Hilton  and  Towers,  Minneapolis,  Minnesota  (Tutorial  TT3). 

Haralick,  R.M.,  1979.  Statistical  and  structural  approaches  to  texture.  Proc.  IEEE,  pp.  786-804. 

Hashimoto,  K.,  1993.  Visual  Servoing:  Real-time  Control  of  Robot  Manipulators  Based  on  Visual 
Sensory  Feedback,  vol.  7  of  Robotics  and  Automated  Systems,  ISBN  981-02-1364-6,  World  Scientific 
Publishing  Co. 

Haton,  J.,  1991.  Le  raisonnement  en  intelligence  artificielle,  ISBN  2-7296-0335-2612293-1, 
InterEditions. 

Jarvis,  R.A.,  1990.  Omniscient  camera  based  outdoor  mobile  robot.  In:  Proc.  Workshop  on  Robotics  in 
Agriculture  and  the  Food  Industry.  1st  IARP,  Avignon-France,  pp.  247-258. 

Kehtarnavaz,  N.,  Grisworld,  N.,  Lee,  J.,  1991.  Visual  control  for  an  autonomous  vehicle  (BART)  —  the 
vehicle  following  problem.  IEEE  Trans.  Vehic.  Technol.  40  (3),  654-662. 

Khadraoui,  D.,  Debain,  C.,  Rouveure,  R.,  Martinet,  P.,  Bonton,  P.,  Gallice,  J.,  1998.  Vision  based 
control  in  driving  assistance  of  agricultural  vehicles.  Int.  J.  Robot.  Res.  17  (10),  1040-1054. 

Khadraoui,  D.,  Motyl,  G.,  Martinet,  P.,  Gallice,  J.,  Chaumette,  F.,  1996.  Visual  servoing  in  robotics 
scheme  using  a  camera/laser-stripe  sensor.  IEEE  Trans.  Robot.  Automat.  12  (5),  743-750. 

Klassen,  N.D.,  Wilson,  R.,  Wilson,  J.,  1994.  Guidance  systems  for  agricultural  vehicles.  In:  Proc.  12th 
World  Cong,  on  Agricultural  Engineering,  International  Commission  of  Agricultural  Engineering, 
Milano,  Italia,  pp.  1136-1142. 

Liu,  W.,  1994.  Detection  par  mesure  de  non  stationarite  locale:  application  a  la  segmentation  de  signaux 
multi-dimensionnels.  PhD  thesis,  Insa,  Lyon,  France. 


C.  Debain  et  al.  /  Computers  and  Electronics  in  Agriculture  22  (2000)  29-51 


Marchant,  J.,  Hague,  T.,  Tillet,  N.,  Sanchez,  J.,  1997.  Research  on  autonomous  vehicle  for  precise  plant 
treatments.  In:  Proc.  Int.  Workshop  on  Robotics  and  Automated  Machinery  for  bio-production, 
Valencia,  Spain,  pp.  237-242. 

Martinet,  P.,  Bonton,  P.,  Gallice,  J.,  Berducat,  M.,  Debain,  C.,  Rouveure,  R.,  1998.  Automatic  guided 
vehicles  in  agricultural  and  green  space  fields.  In:  Proc.  4th  French-Israeli  Symp.  on  Robotics. 
FIR’98.  Besanqon,  France,  pp.  87-92. 

Max,  J.,  1985.  Methodes  et  techniques  de  traitement  du  signal  et  application  aux  mesures  physiques, 
ISBN  2-225-80470-2,  Masson,  Milan. 

Nobutaka,  I.,  1990.  Agricultural  robots  applications  in  japan.  In:  Proc.  IARP  Workshop  on  Robotics 
in  Agriculture  and  Food  Industry,  Avignon,  France,  pp.  35-44. 

Ollis,  M.,  Stentz,  A.,  1996.  First  results  in  vision  based  crop  line  tracking.  In:  Proc.  IEEE  Int.  Conf.  on 
Robotics  and  Automation,  vol.  1.  Minneapolis,  Minnesota,  USA,  pp.  951-956. 

Ollis,  M.,  Stentz,  A.,  1997.  Vision  based  perception  for  an  automated  harvester.  In:  Proc.  IEEE  Int. 
Conf.  on  Intelligent  Robots  and  Systems,  vol.  2.  IROS’97.  Grenoble,  France,  pp.  1838-1844. 

Samson,  C.,  Le  Borgne,  M.,  Espiau,  B.,  1991.  Robot  Control.  The  task  function  approach.  ISBN 
0-19-8538057.  Clarendon  Press,  Oxford. 

Sanchez,  A.,  Marchant,  J.,  1997.  Fast  and  robust  method  for  tracking  crop  rows  using  a  two  point 
through  transform.  In:  Proc.  Int.  Workshop  on  Intelligent  Robotics  and  Automated  Machinery  for 
Bio-production,  Valencia,  Spain,  pp.  28-33. 

Shafer,  G.,  1976.  A  mathematical  theory  of  evidence,  vol.  2702.  Princeton  University  Press,  New  Jersey. 


